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Abstract. Gessel’s famous Bessel determinant formula gives the generating function to enu¬ 
merate permutations without increasing subsequences of a given length. Ekhad and Zeilberger 
recently proposed the challenge to find a suitable generalization to count words of length rn 
in an alphabet consisting of n letters in which each letter appears exactly r times and which 
have no increasing subsequences of length d. 

In this paper we present a generating function for arbitrary values of r expressible as 
multiple integrals of Gessel-type Toeplitz determinants multiplied by the exponentiated cycle 
index polynomial of the symmetric group on r letters. 

1. Introduction and statement of results 


For integers r > 1 and d > 2, let Ad^rin) denote the number of words in the alphabet 
{1, 2,..., n} of length rn in which each letter appears exactly r times and which have no in¬ 
creasing subsequences of length d. As it was pointed out^ by Ekhad and Zeilberger in [EZ], 
the Robinson-Schensted-Knuth (RSK) correspondence (cf. [St]) implies that Ad+i,r{n) can be 
written as a sum over partitions A of weight |A| equal to rn and of length £(A) at most d in the 
following form: 

(1) Ad+i,r{n) = X 

I A|=rn 


(r) 

where fx is the number of standard Young tableaux of shape A and g\ 
standard Young tableaux of shape A whose content is 1’’2’’ ■ ■ - n^. 

For the special case r = 1 (each letter appears exactly once) we have 

(2) Ad+i,i{n) = ^ fl- 


is the number of semi- 
= fx and hence 


|A|=r!, 

t[X)<d 


Gessel [Ge] found the following remarkable generating function for the sequence {Ad+i,i(n)}„: 


( 3 ) 


Ad+l,l{n) 2 n 
^ (n!)2 

•n =n ' ' 




where the entries of the d x d Toeplitz determinant are modified Bessel funetions Im{x) with 
integer parameters, given by the generating function (cf. [DLMF], 10.35.1) 

(4) ^ I^{x)z'^ = exp + z 



Date: June 25, 2015. 

^The formula in [EZ] had a misprint which was found and corrected by Chapuy in [Gh]. 
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For the sake of comparison with the general formulae to be presented, one can write Gessel’s 
generating function (3) in a slightly different but equivalent Toeplitz determinant form; 


(5) 


E Ad+l,l{n) , ,2n 

( n !)2 ' ' 

n=n ' ' 


, det 


where the entries (pm(v,v) are the Laurent series coefficients at z = 0 of the Toeplitz symbol 

OO 


( 6 ) 


ip{v,v;z)= ^ ipmiv,v)z"" =exp(^^+vz^ , 


m— — OQ 


where v denotes the complex conjugate complex-valued book-keeping parameter v and |u| is its 
absolute value. 

Gessel’s Toeplitz determinant formula played a crucial role in the subsequent developments of 
the asymptotic analysis of large permutations, most notably the famous Baik-Deift-Johansson 
theorem [BDJ]. For several related results in this direction see [AD, BOO, ITWl, ITW2, vM, 
TW] and references therein. 

Recently Ekhad and Zeilberger [EZ] proposed the challenge to generalize Gessel’s determinant 
formula (3) for multiple occurrences r > 1. This goal is fully accomplished in the main theorem 
of this paper. 

Before stating the main result, recall that the cycle index polynomial of the symmetric group 
Sr is the multivariate polynomial 


, (-in) 

(7) Cyc,.(ti,...,t^) = - ^ |C^| 

■ |M|=r i=l 

in the indeterminates ti,... ,tr, where the sum is over partitions p of weight |/r| = r, C Sr 
is the conjugacy class consisting of permutations of cycle type p and £{p) stands for the length 
(number of parts) of the partition p. For example, for r = 1, 2,3 the cycle index polynomials are 


Cyci(ti) = ti, 

Cyc2(ti,t2) = ^4 + ^t2, 

Cyc3(ti,t2,i3) = ^4 + ^4*2 + its- 

Note that the cycle index polynomial Gyc^ is a weighted homogeneous polynomial of degree r 
in the natural grading deg(tj) = j. 

Theorem 1. The sequence {Ad+i,r(R)}5))Lo the formal series generating function 
2^d+l,r (j4 I 


(8) E 


(rn)!n! 


luP™ = 


n—0 


JC JC ^<hJ<d 


in the indeterminates v and v, where Lp(v,t]z) is the Toeplitz symbol 


(9) 


ip{v,t;z)= ^ (pm{v,t)z"" = exp I-+'^tj 
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Cyc^(t) is the cycle index polynomial 


£(/i) 


(10) Cyc^(t) = ^ XI n 

|^|=r i—1 

of the symmetric group and dA stands for the area measure in the complex plane. 


Remark 1. The indeterminates v and v are considered independent book-keeping variables in 
terms of which the Toeplitz determinant det (ipi-j{v, t)) and the exponentiated cycle index poly¬ 
nomial e” expanded into formal power series simultaneously and independently. 

The coefficients of the resulting product series 

C30 

det((p,_,(7;,t))e"^Cyc,(*) = X Fk{t)'^v’^v^ 

k,l—0 


are products of weighted homogeneous polynomials in t and t and therefore their integrals with 
respect to multiple planar Gaussian measures in (8) are well-defined. 


Remark 2. Note that^ 


( 11 ) 



r 


= 


, L-t 




where (•,•) is Macdonald’s scalar product on symmetric functions ([Mac], Eq. (4.7)), written in 
terms of the normalized power sum variables ti = pi/i. Therefore we can rephrase (8) as 


( 12 ) 


E Ad+l,rin) 2rn 

irn)\n\ ' ' 
^=0 '' ^ 


det {(fi-jiv, t)), e”" 




Remark 3. The last integral with respect to G in (8) can be replaced by the direct evaluation 

v'' 

(13) tr = — 

r 

as a consequence of the integral identity 

(14) —J P{tr)e~^^~^*’'^ (iA(tj.) = T* ^ for any polynomial P(t). 

The l/r factor in (13) comes from 

(15) [f,]Cyc,(t) = -|C„|=L^ = _. 

This shows that for r = 1 the generating function (8) reduces to Gessel’s Toeplitz determinant 
(5). 


Remark 4. For the case of double occurrences r = 2, there is a quadrature identity 


e 2 dx 


am thankful for M. Bertola for this observation. 
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for polynomials Q{ti,t 2 ), assuming that |'y^/2| < 1. Therefore, for r = 2 our general formula 
(8) reduces to 


Ad+lM'^) _ 1 


{2ny.n\ 


J-oo l<ij<d 


det {ipi-j{v^v^x)) e ^ dx 


(17) 

n—0 

with Toeplitz symbol 

- (v _ \ 

(18) (p{v,v,x\z)= 2Z <fm{v,v,x)z"" = exp i- + vxz +—z"^ j . 

m— — <X) ^ ^ 

By setting v = 1 we can further simplify to 

00 

(19) E 

where 


Ad+1.2{nl^2n ^ 1 


n=0 


(2n)!n! 


7 : 


( 20 ) 


7-00 ^ 

/ \ f 1 

ip{v, x] z) = exp I — I- xz + —z 
\ z 2 


det (ipi-j(v, x)) e ^ dx 


Remark 5. The numbers Ad+i,rin) can be given a standard probabilistic interpretation described 
as follows. Consider the uniform measure on the set of words of length rn in n letters with each 
letter appearing exactly r times, and let Ln,r{w) denote the length of the longest increasing 
subsequence in a random word w, interpreted itself as a random variable. By the definition of 
Ad+i^r(n) we have 

(21) Prob(L„,r(w) <d)= ’ 

and hence a slightly adapted version of (8) can be interpreted as the “Poissonization” of the 
distributions of Ln^r- 


2r 1 

(22) Prob (L^ r(^) ^ d) — 

’ n\ 

n—O 


■ ■ ■ [ det ((pi_j(u,t)) • TT e dA{tj). 

r 

In the probabilistic context (22) may be useful in a subsequent asymptotic analysis of the prob¬ 
abilities Prob (L„^r(R') < d) as n —> 00 and d —> 00 simultaneously, possibly via the Toeplitz- 
Fredholm techniques of [BOO] combined with the de-Poissonization lemma of [J]. 

(r) 

Outline. In Sec. 2 an integral representation of ' is given in terms of the Schur polynomial s\ 
associated to the partition A, which in turn leads to an integral formula for Ad+i,r{'n). Gessel’s 
general Toeplitz determinant for length-restricted Cauchy-Littlewood-type sums over partitions 
is recalled in Sec. 3 and it is employed to complete the proof of Theorem 1. In Sec. 4 we illustrate 
how the abstract generating function can be used in practice to calculate Ad+i^rin) for small 
values of d, r and n by using standard symbolic computer algebra packages. 
Acknowledgements. The author is grateful to Marco Bertola for his interest in this project 
and helpful discussions, and, in particular, for pointing out the alternative scalar product inter¬ 
pretation (12). Thanks also to Doron Zeilberger for his encouragement and kind feedback on 
a previous version of this manuscript, along with a donation to the OEIS in the name of the 
author. The present work is supported in part by the Ponds de recherche du Quebec - Nature 
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2. The Kostka coefficients g ^{^ as integrals of Schur polynomials 

Following the standard approach in the theory of symmetric functions [Mac], one defines the 
Schur polynomials sx G C[ti, ^ 2 , ^ 3 , ■ • ■ ] in the normalized power sum variables t = 
via the Jacobi-Trudi formula 


(23) 


SA(t) = det , 


where the polynomials hk(t) (the complete symmetric polynomials) are obtained from the gen¬ 
erating function 


(24) 


hk{t)z'' = exp j tj 


k=0 




with hk{t) = 0 for negative values of k. The polynomial hk{t) is a weighted homogeneous 
polynomial of degree k in the natural grading 


(25) 


deg(tj) = j. 


Therefore SA(t) is also a weighted homogeneous polynomial of degree |A| and hence it has the 
scaling property 


(26) 


SA (ati,a^f2,a^t3, ■ • ■) = a'^''SA(ti,<2,<3, • ■ • )■ 


It is well-known [Mac] that the number of standard Young tableaux of shape A is given by the 
special evaluation 


(27) 


/a = |A|!sa (1,0,0,...). 


As it is pointed out in [Ch], g^^^ is the Kostka number associated to the pair of partitions A 

and (r"), that is, g^^^^ is the coefficient of m(rn) in the expansion of s\ in the monomial symmetric 
function basis of C[t] (see [Mac]): 

( 28 ) 5 ^’'^ = [xlxl^ ■ ■ ■ xl] sx{ti{x.),t2{x),...), 

where 


(29) 




^ i=i 


are the normalized power sum symmetric polynomials in the variables xi,..., a:„. 

In what follows, the coefficient representation (28) of g^^^ will be rewritten as an r-fold integral 
of the Schur polynomial associated to A in terms of the normalized power sum variables. To 
proceed in this direction we need the following basic but important 

Proposition 1. Let P(ti, ^ 2 , ■ • ■, U) a polynomial in the indeterminates ti, ..., U consider 
the polynomial 


( 30 ) 


P (ti(x),t2(x),... ,u(x)) G C[a;i, ...,Xn] 
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in the indeterminates xi,... ,Xn, where tki'x) are the normalized power sum symmetric polyno¬ 
mials (29) in X. The following integral identity holds: 


(31) [xlx^2 ■ ■ ■ Xu]P (il(x), <2(x), . . . , tr{x)) 



r 


where h^ (mi, ..., 0 ,0,...) is the complete symmetric polynomial of degree r evaluated at 

(ui, ..., 0,0,...) and dA is the area measure in the complex plane. 

Remark 6. A multiple integral formula for g^^^ involving h'f was found in [Ch], expressed in 
terms of the original indeterminates Xj. Our identity (31), expressed in terms of the normalized 
power sum symmetric polynomials tfc(x), has essentially the same combinatorial content as the 
one presented in [Ch]. As it will be clear below, our choice of coordinates turns out to be more 
advantageous in finding a generating function for the numbers ,,( 71 ). 


Proof of Lemma 1. For any r-tuple of non-negative integers ki,k 2 ,... ,kr let 
(32) ■=[xW2---Xn]tl{x)'^^ 

The exponential generating function of the numbers can be calculated as follows: 


(33) 

00 r kj 

E “E..,fe.nTT = 

■•<] ] 


Ti 

0 

II 

F 

ki,.. 

(34) 


■ ■ Xn] exp 

(35) 

= [xlxl^ • ■ 

n 

' • <] n' 


(36) 

(37) 
Since 

(38) 

we have 


(39) 


1 


E n 


kj\ 


Vi=i 


Ef^ 

J 




Uj 


Uo Ur 

— — 0 0 

2 r 




/c 


= (jTr (ui, 

- [ z'=z'e-l"l'dA(z) = 6kik\ 

^ Jc 

(hr (wi, y,.. . , y, 0, 0,..e“l“^l"dA(Mj) 


= ^kll-Ar = [^1^2 • • • <]^l(x)''' • • • ir(x)'=’' 

for all fci, k 2 , ■. ■, kr, and hence (31) follows by taking linear combinations. □ 
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Corollary 1. For a partition of weight |A| = rn we have 
(40) 


^ J^sx{ti,...,tr,0,...)(^hr J^e ^*^^^dA(tj) 


Note that for a pair of weighted homogeneous polynomials P and Q, 

(41) —/•••/ P{ti, ■ ■ ■ ,tr)Q{ti, ■ ■ ■ ,tr)Y\ dA{tj) = Q unless degP = degQ. 

TT’' Jc ^ JC 

r 

Since s\ and are weighted homogeneous polynomials with deg(sA) = |A| and deg(/i") = rm 
we have 

(42) 

^ ^ SA(tl, . . . , tr, 0 , . . . ) (^r (il, y, ■ • ■ , 0 , . . . ) ) e"l*^ l"dA(tj) = 6 \x\,rmfx- 

r 

Remark 7. For example, for r = 1,2,3 we have 

hi (iti, 0 ,...) = ui 

l. f '“ 2 n ^ 42,4 

h2 = -Ui + -U2 

/ U 2 U 3 nIoI, 1 

(^Ml, y, y, 0 , . . . J = g«l + 2 ^l '“2 + 3 ^ 3 . 

The general formula is (cf. [Mac], (2,14’)) 


(43) 




M=r ^ i=l 


where is the standard constant factor 


(44) 


z^i = • m^l 

i>l 


in which rrii = rni{fi) is number of parts of the partition fi equal to i. Note that 

r\ , , 


(45) 


\Cf.\ = 


where is the conjugacy class of permutations of cycle type /i in the symmetric group Sr- 
Hence hr (ui, ..., 0,0,...) can be written alternatively as the cycle index polynomial of 

Sr-. 


t{X) 


(46) hr (ui, y,. .., y,0,. .^ ICaI P UAi = Cyc^(Mi,M2,- ■ ■,Ur). 

’’ |A|=r i=l 

By taking into account (42) and (46) we obtain the following 

Corollary 2. 

(47) 

1 
tt' 


^ Jc Jc_ J^i 


^Ox'’ I A| = rn 
0 otherwise. 
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By the RSK formula this leads to an integral representation of Ad+i^rin): 
Lemma 1. The following identity holds for all meaningful values of d,r and n: 


(48) ^ J_ 

(rn)!n! tt*' 


/C JC 


^ sx{v,0,...)s\{ti,...,tr,0,...) 


\X\—rn 

.l{X)<d 


X e 


F’'Cyc^(ti,....ir) TTp-hil 


n® dA{tj). 

i=i 


Proof. By using the elementary identity 
(49) 

we obtain 


= sa(^^,0 ,0, ...) 


(50) 

Ad + l , r { n ) ii2rn 
(rn)!n! ' ' 




(51) 

f P ) 

— JjLajlM . 3 x -rn 

1 • n! 

\ X\=rn ' ' 





t ( X)<d 




(52) 

= Y sa(?i,0, ...)• f ■■ 

• / sxih ,. 
Jc 

.., 0,.. 

)g'«”Cyc,,(ti 




(53) =- 


'c Jc 


^ Sx{v,0,...)sx{ti,...,tr,0,...) 


I Al^rn 
l{X)<d 


„D’’Cyc^(tl,...,tr) TT 


ne-l‘^1 dAit,). 

i=i 


□ 


3. Gessel’s Toeplitz determinant identity and the proof of the main theorem 

Recall Gessel’s determinant formula [Ge]: the length-restricted Cauchy-Littlewood-type sum 
of products of Schur functions can be written as 

(54) SA(t)sA(s)= det (v3i-i(t,s)) 

e{x)<d - 

where t = {ti,t 2 , ■ ■ ■),s = (si, S 2 , • ■ •) and the symbol (f associated to the above Toeplitz deter¬ 
minant is 

( OO OO \ OO 

k=l k=l / m= —OO 

This “master identity”, combined with the integral representation for Ad+i^rin) is sufficient to 
prove the main result of this paper. 


Proof of Theorem 1. Gessel’s determinantal formula (54) with 

(56) t = (ti,..., tj., 0,, 0 ...) and s = (u, 0,0,...) 
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gives 

(57) sx{v,0,...)sx{ti,...,tr,0,...) = det ((/5i_j('y,t)) 

£{X)<d 

with Toeplitz symbol 


(58) 




OO 

y2 = exp 

m — — oc 



Therefore 


(59) 


(60) 



r 


(61) 



y2 Sx{v,0,...)sx{tl,...,tr,0,...) 

£{X)<d 

_|A|=rn 


1=1 


where in the last step we used the simple observation that (47) implies that a partition A may 
give a non-trivial contribution to the integral only if its weight |A| is divisible by r. The proof is 
concluded by comparing (61) with the integral representation (48) of Ad+i,r{n). □ 


4. Calculation of Ad + i , r { n ) using symbolic computer algebra 

The purpose of this section is to illustrate that the generating function (8) can be used to 
calculate Ad+i.r{n) for small values of d,r and n using basic symbolic computer algebra. There 
was no attempt made to optimize the algorithms used in the symbolic computation, our goal was 
simply to demonstrate that the generating function is not only a compact and abstract formula, 
but it can also be put to work in practice. This also provides numerical evidence that the series 
(8) does give the integer numbers it is supposed to, which, after all, is the most rewarding part 
of finding a generating function. 

Below we use the short-hand notation = {ti,... ,tr)- It is easy to see that the Laurent 
series coefficients of the Toeplitz symbol (p{v,tr',z) are 


(62) 


^r) — ^ ^ ^A:+77i(tr) — 0, 1, 2, . . . 


OO 




k—m 




(63) 


m = 1,2,..., 
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where hk{tr) are the complete symmetric polyomials in t^. We can define the truncated Toeplitz 


entries 


(64) 

rm 

(65) 

{N) 

so that 

(66) 

( 


rN 


rN 


)— 

k\ 


k—m 


= '^ hk-m{tr)^ 771=1,2,... 


For small values of d the Toeplitz determinant with O (7;’’^“''^)-truncated entries can be evalu¬ 
ated, and it can be further simplified by eliminating the terms of O We note that the 

evaluation of the determinant can be done much more efficiently by employing the division-free 
algorithm suggested in [ESZ]. 

The second factor in (8) e'"’' has the truncated expansion in v of the form 

N 

(67) Tj:^\ti,i2/2,...,ir/r-,v) = ^< 2 / 2 ,..., trA, 0, 0)'=^ 

k^O 

for which 

(68) e"” Cyc,Cti,-rtr) ^ U/r- v) + O . 

Therefore by combining the truncated factors of the integrand we find 


(69) 


1 


Jc Jc 


T 

det dA{tj) 

's 2 - 7 d \. J 


J = 1 


N 


— ^ ^d+l,r{n) |^,|2r-7i q ^ |2rAr-|-2r 


n=0 


(rn)!n! 


)■ 


The computation of area integrals can be implemented easily by using polar coordinates 


(70) 


[ F{t)e 1*1 dA{t) = 2 /" (f F{ru) 

Jc Jo 2777 J\u\ = l 


du 

u 


eJ rdr 


and evaluate the contour integrals using a residue calculation at u = 0 {F(u) is a polynomial in 
u for any variable u in our setting. 

This simple method was implemented as a Maple code and we obtained the values shown 
in Tables 1-3 These all coincide with the known values presented in various appendices of the 
journal entry [EZ]. 


d\n 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

2 

1 

6 

43 

352 

3114 

29004 

280221 

2782476 

28221784 

291138856 

3 

1 

6 

90 

1879 

47024 

1331664 

41250519 

1367533365 

47808569835 

1744233181074 

4 

1 

6 

90 

2520 

102011 

5176504 

307027744 

20472135280 

1496594831506 

117857270562568 


Table 1. The calculated values of Ad+i, 2 (n) 
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d\n 

1 

2 

3 

4 

5 

1 

1 

1 

I 

1 

1 

2 

1 

20 

374 

8124 

190893 

3 

1 

20 

1680 

173891 

21347262 

4 

1 

20 

1680 

369600 

117392909 


Table 2. The calculated values of Ad+i^sin) 


d\n 

1 

2 

3 

4 

5 

1 

1 

1 

1 

1 

1 

2 

1 

70 

3199 

173860 

10203181 

3 

1 

70 

34650 

16140983 

8854463421 

4 

1 

70 

34650 

63063000 

142951955371 


Table 3. The calculated values of Arf+i_ 4 (n) 


5. Conclusion and outlook 

In this paper we presented the generating function (8) to solve the challenge posed by Ekhad 
and Zeilberger [EZ] to generalize Gessel’s Bessel determinant formula (3) for arbitrary values of 
the number of occurrences r. The key step in constructing these generating functions was to 
express the coefficient as a use Gessel’s “master identity” to sum up the resulting Cauchy- 
Littlewood-type terms under the integrals. 

We expect that the general formula (8) may be useful to find the associated linear differential 
equation with polynomial coefficients for the generating function, or equivalently, to find the 
recurrence relation with polynomial coefficients for Ad+i^rin) for arbitrary values of r in a closed 
form. 

Moreover, one can easily construct generating functions analogous to (8) for words with 
different types of multiple occurrences in its letters. 

It may also be interesting to investigate the asymptotic distribution obtained from a fine- 
tuned scaling limit of the probability Prob [Ln^r{w) < d) as n, d —^ oo. It is then natural to ask 
which universality class the limiting probability density belongs to, either a to a known one or 
to one of a new type that has not yet been characterized. 
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